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DESCRIPTION 

5 

BACKGROUND OF THE INVENTION 
Fi.e2d of the JnirentJLon 

10 The present invention generally relates to eye gaze 

Jf| trackers and, more particularly, to techniques for improving 

W accuracy degraded by ambient light noise while maintaining 

Hi safe IR levels output by the illuminator. 

Ql 15 Descjr±pt±on of the Related Azrt 

Cl 

Wi The purpose of eye gaze trackers, also called eye 

trackers, is to determine where an individual is looking. 

O The primary use of the technology is as an input device for 

20 human-computer interaction. In such a capacity, eye trackers 

enable the computer to determine where on the computer 
screen the individual is looking. Since software controls 
the content of the display, it can correlate eye gaze 
information with the semantics of the program. This enables 
25 many different applications. For example, eye trackers can 

be used by disabled persons as the primary input device, 
replacing both the mouse and the keyboard. Eye trackers have 
been used for various types of research, such as determining 
how people evaluate and comprehend text and other visually 
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represented information. Eye trackers can also be used to 
train individuals who must interact with computer screens in 
certain ways, such as air traffic controllers, nuclear 
energy plant operators, security personnel, etc. 
5 The most effective and common eye tracking technology 

exploits the "bright-eye" effect. The bright-eye effect is 
familiar to most people as the glowing red pupils observed 
in photographs of people taken with a flash that is mounted 
near the camera lens. In the case of eye trackers, the eye 
10 is illuminated with infrared light, which is not visible to 

the human eye. An infrared (IR) camera can easily detect the 
infrared light re-emitted by the retina. It can also detect 
the even brighter primary reflection of the infrared 
illuminator off of the front surface of the eye. The 
15 relative position of the primary reflection to the large 

circle caused by the light re-emitted by the retina (the 
bright-eye effect) can be used to determine the direction of 
gaze. This information, combined with the relative positions 
lJI of the camera, the eyes, and the computer display, can be 

r"^ 20 used to compute where on the computer screen the user is 

looking . 

Eye trackers based on the bright-eye effect are highly 
effective and further improvements in accuracy are 
unwarranted. This is because the angular errors are 

25 presently smaller than the angle of foveation. Within the 

angle of foveation, it is not possible to determine where 
someone is looking because all imagery falls on the high 
resolution part of the retina, called the fovea, and eye 
movement is unnecessary for visual interpretation. 

30 However, despite the effectiveness of infrared 
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bright-eye based eye tracking technology, the industry is 
highly motivated to abandon it and develop alternative 
approaches. This is deemed necessary because the 
infrared-based technology is not usable in environments with 
5 ambient sunlight, such as sunlit rooms, many public spaces, 

and the outdoors. To avoid raising concerns about potential 
eye damage, the amount of infrared radiation emitted by the 
illuminators is set to considerably less than that present 
in normal sunlight. This makes it difficult to identify the 
10 location of the bright eye and the primary reflection of the 

^l illuminator due to ambient IR reflections. This, in turn, 

diminishes the ability to compute the direction of eye gaze. 

m 
01 

41 SUMMARY OF THE INVENTION 

1 15 

w 

5 The present invention is directed to techniques for 

. improving accuracy in the signal to noise ratio of an eye 

rU tracker signal degraded by ambient light noise. It enables 

~I the effective use of bright-eye based eye tracking 

20 technology in a wider range of environments, including those 

with high levels of ambient infrared radiation. Of course 
one way in which to do this would be to increase the 
intensity of the IR illuminator to overcome the ambient 
sunlight. However, this solution is not viable since 
25 increased IR radiation has associated health risks. 

Instead, the invention exploits the observation that 
the intensity of sunlight and its constituent wavelengths of 
light, such as infrared radiation, do not vary rapidly. 
During the inter-frame interval of video cameras (typically 
30 l/30th of a second), the level of ambient infrared radiation 
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can be considered nearly constant. 

The invention modulates the intensity of the 
illuminator with respect to time so that the illuminator 
signal may be extracted from the nearly constant ambient 
5 infrared radiation. The modulation of the illuminator is 

synchronized with the control of the camera/digitizing 
system to eliminate the need for pixel by pixel demodulation 
circuits. Several embodiments are disclosed for extracting 
the ambient IR (i.e., the noise) from the IR signal. In the 
10 first embodiment, the modulation of the IR illuminator is 

O synchronized with each frame of the camera such that the 

Si illuminator alternates between on and off with each 

Jtf subsequent frame. A video frame grabber digitizes and 

42 captures each frame. If one considers a sequence of such 

15 frames, then the image captured in the first frame contains 

s both the illuminator signal and the ambient radiation 

information. The image captured in the second frame contains 
fil only the ambient radiation information. By subtracting, 

~: pixel-by-pixel, the second frame from the first frame, a new 

1=1 

M 20 image is formed that contains only the information from the 

illuminator signal. The resulting image can then be used by 
the conventional eye tracker system to compute the direction 
of eye gaze even in the presence of an ambient IR source. 
Other embodiments or variations are also disclosed for 
25 reducing ambient IR noise. 



BRIEF DESCRIPTION OF THE DRAWINGS 
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The foregoing and other objects, aspects and advantages 



YOR9-2000-0 699-US1 




01 

pi 

III 

5 15 

s \ 
O 

H 20 



will be better understood from the following detailed 

description of a preferred embodiment of the invention with 

reference to the drawings, in which: 

Figure 1 is a diagram showing the basic set up of the 

eye gaze control system according to the present invention; 
Figure 2 is a diagram illustrating how ambient IR 

radiation effects the eye gaze control systems- 
Figure 3A is a diagram illustrating IR noise mixed with 

the reflection signal when the illuminator is turned on for 

a first frames- 
Figure 3B is a diagram illustrating just the noise 

acquired by turning the illuminator off for a second frame; 
Figure 3C is a diagram illustrating the reflection 

signal having an improved S/N ratio by subtracting the 

second frame from the first frame; 

Figure 4 is a diagram illustrating improving the S/N 

ratio by synchronizing the illuminator modulation for 

interleaved raster fields; 

Figure 5 is a diagram illustrating improving the S/N 

ratio by synchronizing the illuminator with the even and odd 

horizontal pixels; and 

Figure 6 is a diagram illustrating improving the S/N 

ratio by illuminating odd and even pixels in alternating 

interleaved raster fields forming a checkerboard pattern. 



Referring now to the drawings, and more particularly to 



DETAILED DESCRIPTION OF A PREFERRED 



EMBODIMENT OF THE INVENTION 
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Figure 1 there is shown a typical set up for the present 
invention. A display monitor 10 is connected to a computer 
12 and positioned in front of a user 14. Traditional input 
devices such as a keyboard 16 or mouse (not shown) may also 
5 be present. However, in certain situations, the user may 

have physical constraints that render them unable to use 
traditional input devices. Therefore, the present invention 
provides an alternative to these traditional devices and 
would be useful for any individual capable of moving his or 
10 her eyes, including a quadriplegic or similarly disabled 

person. Although the user 14 is shown in a sitting position, 
-ii the user could of course be lying down with the display 10 

5? and eye tracker 18 positioned overhead or visible through an 

HI arrangement of mirrors. 

g| 15 An eye gaze tracker 18 is mounted and aimed such that 

the user's eyes 22 are in its field of vision 20. The eye is 
illuminated with infrared light. The tracker 18 detects the 
infrared light re-emitted by the retina. This information, 
b|| combined with the relative positions of the tracker 18, the 

J=f 20 eyes 22, and the computer display 10, can be used to compute 

where on the computer screen the user 14 is looking 24. 

As shown in Figure 2, the computer 12 outputs a display 
signal 40 to control the images on the display 10. The eye 
gaze tracker 18 comprises an illuminator portion 30 and a 
25 camera 32. As shown, the illuminator 30 comprises a ring of 

IR sources around the camera 32 in the center of the ring. 
This ring-type arrangement is shown for example in U.S. 
Patent 5,016,282 to Tomono et al. However, there are many 
arrangements of illuminator and camera that may be suitable 
30 for this application. The computer 12 supplies an 
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illuminator signal 42 to control the output of the 
illuminator 30. The illuminator 30 illuminates the user's 
eye with a beam in IR light 20. The IR camera 32 can easily 
detect the infrared light re-emitted by the retina. It can 
5 also detect the even brighter primary reflection 34 of the 

infrared illuminator 30 off of the front surface of the eye. 
The reflection signal 44 from the camera 32 is fed back to 
the computer 12 for processing. However, as previously 
noted, in the presence of another IR light source, such as 
10 ambient sunlight 36, the reflection signal 44 includes not 

□ only information owed to the reflected illuminator light 34, 

but also noise caused by the ambient light 36.. While the 
ffl sunlight 36 is shown directly entering the camera 32, it 

will be appreciated by those skilled in the art that the 
Bl 15 ambient light picked-up by the camera 32 may also be 

■^"^ sunlight or light from other sources reflected off of the 

CI subject 14, walls, ceilings, other objects in the room. 

^5 Therefore, if there is appreciable ambient light, the 

41 signal-to-noise (S/N) will be low and the computer 12 may 

20 have difficulties in accurately detecting the position of 

the user's gaze position on the display 10. 

The first embodiment of the present invention, exploits 
the observation that the intensity of sunlight and its 
constituent wavelengths of light, such as infrared 
25 radiation, do not vary rapidly. During the inter-frame 

interval of the camera 32 (typically l/30th of a second); 
the level of ambient infrared radiation can be considered 
nearly constant. Therefore, the computer modulates the 
intensity of the illuminator 30 with respect to time. In 
30 this case, the modulation of the illuminator signal 42 is 
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synchronized with each frame of the camera 32 such that the 
illuminator 30 alternates between on and off with each 
subsequent frame. A video frame grabber 4 6 digitizes and 
captures each frame. If one considers a sequence of such 
frames, then the image captured in the first frame contains 
both the illuminator signal and the ambient radiation 
information. The image captured in the second frame contains 
only the ambient radiation information. By subtracting, 
pixel-by-pixel, the second frame from the first frame, a new 
image is formed that contains only the information from the 
illuminator signal. The resulting image can then be used by 
the conventional eye tracker system to compute the direction 
of eye gaze. The process would then be repeated starting 
with the third frame. The resulting system would yield 15 
eye gaze direction computations per second with a typical 
camera and frame grabber system. 

Still referring to Figure 2, this process is 
illustrated in Figures 3A-C. Figure 3A represents the first 
frame in a sequence of frames. During this first frame, the 
illuminator 30 is turned on and is illuminating the user's 
eye with IR light. Due to ambient IR light in the room, 
reflection signal 44 comprises both the desired reflection 
signal 34, as well as the noise caused by the ambient light 
36. In the second frame shown in Figure 3B, the illuminator 
30 is turned off and the camera only sees the ambient light 
or reflections caused by the ambient light 36. Therefore, 
the reflection signal 44 only contains the noise as 
illustrated in Figure 3B. If a pixel by pixel subtraction is 
carried out, subtracting the image of Figure 3B from the 
image of Figure 3A, the resultant image, as shown in Figure 
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3C will be that caused by the illuminator 30 which is 
substantially devoid of the ambient noise and can be used to 
compute the direction of eye gaze. 

The embodiment described above is limited by two 
5 factors. The first is the combined signal to noise ratio of 

the infrared video camera 32 and the frame digitizer 46. 
This signal to noise ratio must be less than the signal to 
noise ratio of the illuminator signal to the ambient 
radiation. This limitation applies to all embodiments and is 
10 the fundamental constraint on the range of environments in 

□ which the system can be used. 

The second factor is temporal resolution. As noted 

ff^ above, the first embodiment produces 15 eye gaze direction 

111 

J! computations per second. This rate can be effectively 

15 doubled by subtracting each subsequent frame and taking the 

absolute value of the result. If the "absolute value" 
operator is not available, then it can be approximated by 

ry adjusting the manner in which subtraction is performed. 

if i 

^: Consider the following example: first, assume that the 

f^b 20 illuminator is turned on during even numbered frames and off 

during odd numbered frames. At time 1, the first output 
image, Oi, is computed by subtracting frame 1, f^, from 
frame 0, fo- Thus, Oi = fo - fi. At time 2, the order of 
subtraction must be changed to avoid negative image values: 
25 02 f2~fi- At time 3, the original subtraction order is 

restored: 03 = fs - fa- The process continues indefinitely as 
follows: 04 = f^ - f3, 05 = f4 - fs, Og = fe - fs, and so on. 
This can be expressed as o^ = | f n - fn-i|- 

In this manner, up to 30 eye gaze direction 
30 computations per second are possible with typical camera and 
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frame grabber systems. If a one frame period of delay is 
acceptable, temporal second order techniques for estimating 
noise or signal plus noise is possible. For example, at time 
2, ol would be produced as follows: ol = | f 1 - (fO + f2)/2|. 
5 This expression can be more generally written as o^ = I f n ~ 

(fn-l + fn.l)/2| . 

If even greater temporal resolution is required, it may 
be acquired at the expense of spatial resolution by 
synchronizing the illuminator 30 with the fields instead of 
10 the frames. To reduce the appearance of flicker most video 

camera standards use interleaving- As shown in Figure 4 
interleaving first scans the even numbered horizontal lines 
S| of a frame and then the odd numbered lines. In this manner 

yl the full height of the frame is scanned twice per frame, or 

gl 15 typically once every l/60th of a second. Each half of a 

ttl frame scanned in this manner is called a "field" and each 

field has half the vertical resolution of a frame. In this 
case, the illuminator 30 is turned on during the scan of 
[| field 1 and turned off during the scan of field 2. Thus 

" 20 field 1 contains the actual reflection signal mixed with the 

noise signal and field 2 contains only the noise signal due 
to the ambient light. Subtracting raster lines in field 2 
from adjacent raster lines in field 1 nearly eliminates the 
noise signal, 

25 As shown in Figure 5, in the third embodiment, the 

computer synchronizes the illuminator 30 with the even and 
odd horizontal pixels. For example, the illuminator would be 
on for all even numbered horizontal pixels and off for the 
odd numbered horizontal pixels. This would effectively form 

30 alternating vertical stripes consisting of signal and noise 
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or just noise information. The illuminator signal would be 
extracted by subtracting adjacent pixels from each other and 
taking the absolute value. Naturally, this modulation scheme 
would require an illuminator 30 capable of turning on and 
off many hundreds of times faster than required for the 
other schemes. This approach could be used with frames or 
fields. 

As shown in Figure 6, the second and third modulation 
techniques shown in Figures 4 and 5 can also be combined to 
yield a checkerboard pattern of noise pixels and signal plus 
noise pixels with adjacent pixels being subtracted to yield 
a reflection signal having improved S/N characteristics. 

Spatial and temporal second order techniques as 
described above could also be used for noise and signal plus 
noise estimation for any of the above embodiments. 

In addition, this invention is preferably embodied in 
software stored in any suitable machine readable medium such 
as magnetic or optical disk, network server, etc., and 
intended to be run of course on a computer equipped with the 
proper hardware including an eye gaze tracker and display. 

While the invention has been described in terms of a 
several preferred embodiments, those skilled in the art will 
recognize that the invention can be practiced with 
modification within the spirit and scope of the appended 
claims . 
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